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Abstract 

The present paper outlines two types of discriminant analysis, predictive discriminant analysis 
(PDA) and descriptive discriminant analysis (DDA). Important differences between PDA and 
DDA are introduced and discussed using a heuristic data set, specifically indicating the potions of 
the SPSS output relevant to each of the types of discriminant analysis. The importance of 
knowing the distinguishing features of PDA and DDA is highlighted using an example of stepwise 
discriminant analysis. 
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Two Types of Discriminant Analysis: NOT Six of One and Half a Dozen of the Other 
In areas of behavioral science such as psychology and education, researchers tend to look 
at how a number of variables (e.g., SES, IQ, and parent involvement) affect a specific outcome 
(e.g., academic achievement). It is rare in these areas of research to find efforts looking at only 
one variable at a time. Huberty (1994) reminds us that 

empirical research in nearly every discipline is rarely confined to the study of a 
single response variable, a characteristic or attribute or trait on which the 
researcher obtains scores or responses for a collection of units. Data sets 
typically involve measures on a number of variables, and it may be desirable to 
consider... all the variables simultaneously, (pp. 26-27) 

This model of reality makes it important for researchers to understand the concepts of 
multivariate statistics (Thompson, 1994b). Thompson (1994b) argued that multivariate statistics 
“limit the inflation of Type I ‘experimentwise’ error rates” (p. 9). A lower Type I error rate helps 
ensure correct conclusions, which is important to sustaining the life of research efforts 
(Thompson, 1 994b). More importantly, multivariate analyses also “best honor the reality to 
which the researcher is purportedly trying to generalize” (Thompson, 1994b, p. 12), that is, a 
reality in which most effects are multiply caused and most causes have multiple effects. 

Multivariate analyses can be conceptualized in a number of ways. While Knapp (1978), in 
his discussion of the general linear model, did conclude that “all of the commonly encountered 
parametric tests of significance can be treated as special cases of canonical correlation analysis” 
(p. 410), each multivariate technique is unique in its own way. It is important for researchers to 
understand the similarities and the differences among multivariate analyses. Techniques such as 
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factor analysis are used when interested in clustering variables, or, in other words, when we want 
to study individual differences on a single group of variables (Huberty, 1994). Klecka (1980) 
describes a technique, the focus of this paper, which does not look at individual cases, but at data 
cases that comes from two or more mutually exclusive groups: discriminant analysis. 

While univariate techniques are still the most popular analyses, the use of multivariate 
techniques is on the rise in research journals (Elmore & Woehlke, 1988). For example, Emmons, 
Stallings and Layne (1990) studied 16 years of research reports in three journals, and found that 
the multivariate characteristic of the social science research environment with its 
many confounding or intervening variables has been addressed through the trend 
toward increased use of multivariate analysis of variance and covariance, multiple 
regression, and multiple correlation, (p. 14) 

Similarly, Grimm and Yarnold (1995) recently noted that, “in the last 20 years, the use of 
multivariate statistics has become commonplace. Indeed, it is difficult to find empirically based 
articles that do not use one or another multivariate analysis” (p. vii). 

One unfortunate impetus for increased use of multivariate methods is the new point-and- 
click microcomputer software environment in which researchers now work. Computer point-and- 
click software makes blind reliance of its use very easy. Many researchers tend to grab output 
and analyze data haphazardly, with no real understanding of these sophisticated multivariate 
techniques; discriminant analysis is no stranger to this phenomenon. 

The present paper explains some of the basic principles and concepts underlying 
discriminant analysis by differentiating two types of discriminant analysis (DA), predictive 
discriminant analysis (PDA) and descriptive discriminant analysis (DDA). These differences will 
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be clarified and concretely illustrated using a heuristic data set taken from Holzinger and 
Swineford (1939). Holzinger and Swineford (1939) collected data on 301 seventh and eighth 
graders. The data represent scores on a number of performance tests ranging from reading 
comprehension to mathematical reasoning. 

The importance of understanding the differences between PDA and DDA (see Huberty, 
1994, and Thompson, 1995a) will be highlighted by applying the distinctions to specific research 
questions and situations. The paper cautions against blind reliance on statistical software 
packages. The package illustrated in this paper is SPSS, although these issues apply to other 
software as well. 

Discriminant analysis is one multivariate method available “[w]hen groups of units are 
known in advance and the purpose of the research is either to describe group differences or to 
predict group membership on the basis of response variable measures” (Huberty, 1994, p. 28). 
DA is a procedure which “identifies boundaries between groups of objects, the boundaries being 
defined in terms of those variable characteristics which distinguish or discriminate the objects...” 
(Kachigan, 1982, p. 216). 

The units of analysis in discriminant analysis, whether we are studying people, animals or 
political parties, need to be from two or more mutually exclusive groups, and each case needs to 
have corresponding values on some set of response variables, which are measured on an interval 
or ratio level of measurement (Klecka, 1980). The grouping variable, on the other hand, is 
categorical in nature. It is DA’s ability to look at this nominal variable that make it desirable in 
behavioral science research. Multiple regression is inappropriate in that the dependent variable in 
this prediction model needs to be continuous in nature. As in other analyses, the response 
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variables are combined in a linear fashion to describe groups or to predict group membership 
(Huberty, 1994). 

The use of DA carries with it some assumptions. These assumptions are fully outlined in 
Klecka (1980) and Huberty (1994). One assumption is that each group is drawn from a 
population with multivariate normal distributions of the response variables, and no response 
variable can be a linear combination of any other variables. For example, it would be 
impermissible to use dollars earned per month last year in an equation along with dollars earned 
last year and number of months worked last year. While it is beyond the scope of the current 
paper to discuss the intricacies of parameter estimation in DA, the interested reader is 
encouraged to refer to Klecka and to Huberty for a detailed summary. Ashcraft (1998) also 
summarizes these issues and provides a nice short SPSS for Windows program that implements a 
graphical test of multivariate normality; the program is reproduced here in Appendix A. The rest 
of this paper focuses on describing and contrasting the two types of discriminant analysis, PDA 
and DDA. 

To repeat, their are two separate reasons why a researcher would use DA. One is to 
describe group differences, and one is to predict group membership While both fall under the 
umbrella of discriminant analysis, the two types of DA differ in focus, analysis, and interpretation. 
The two analyses also differ historically. Huberty (1994) noted that PDA (and the associated 
linear classification function (LCF)) was conceptualized first. DDA (and the associated linear 
discriminant function (LDF)) came later in order to explain effects found in MANOVA. LDF’s 
and DDA’s relation to MANOVA will be covered later. 

Prior to discussing PDA, however, an important note needs mentioning. Table 1 includes 
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a sample output from an SPSS run. The computer was given a number of variables from 
Holzinger and Swineford (1939), and told to run a discriminant analysis (sample syntax can be 
found in Appendix B). Note that the syntax does not indicate “PDA” or “DDA”. SPSS gives you 
everything you need to do either PDA or DDA, but does not differentiate the relevant aspects of 
the results as regards these analyses. You, the researcher are left to discern the relevant 
information from the irrelevant information given your research question. This is a good example 
of the need to fully understand the analyses one asks of computers, and what information needs to 
be focused on depending on the research question. Of course, since many researchers blindly use 
computers, and have blind faith in software, many researchers not seeing SPSS distinguish PDA 
from DDA, wrongly presume that all printed results are relevant to both analyses. 



Insert Table 1 about here 



Predictive Discriminant Analysis 

As previously indicated, PDA is used when researchers wish to predict a data case’s group 
membership based on responses on a number of variables of interest. While data cases could be 
things, people, or animals, this paper, using data on seventh and eighth graders, will refer to data 
cases as people. Klecka (1980) describes PDA as “the process by which a decision is made that a 
specific case ‘belongs to’ or most closely resembles’ one particular group” (p. 42). A research 
question posed for the Holzinger and Swineford (1939) data, therefore, might be, “Given student 
responses/scores on performance measures, can we predict their grade placement?” 

In PDA, response variables serve as the predictor, or independent variables, and the 
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grouping variable is the dependent or criterion variable. Data are collected on a number of 
response measures, and these variables are used to form a classification “rule” (also called a 
“function,” “factor,” or an “equation”, mainly to confuse the graduate students) which will help 
predict membership for a unit (person). According to Huberty (1994), a classification rule can 
take on various forms: “...that of a composite of the predictor measures;... that of a probability of 
population membership;... [and] that of a distance between two points” (p. 40). 

With regard to distance, assume we have three measures, and consider two people’s 
scores on these three measures as points in a three-dimensional space. We can look at three types 
of distances in this space “(1) unit to unit... (2) centroid to centroid,[see Ashcraft, 1998, for an 
explanation] ...and (3) unit to centroid” (Huberty, 1994, p. 44). A unit is one person’s vector of 
scores, and the centroid is the Cartesian coordinate defined by the means across all the response 
variables. A decision on group membership is based on which centroid to which a unit falls 
closest. 

Group prediction can also be accomplished through the use of a linear classification 
function, or LCF. An LCF is a linear combination of the measures of interest, and there are as 
many LCF’s as there are groups to be predicted (Huberty, Wisenbaker, & Smith, 1987). As in 
other linear composites of response variables (i.e., linear regression), weights are applied to each 
of the independent predictor variables, and “[t]he weights in these composites are products of 
sample mean response variable vectors and the inverse of the pooled within-groups covariance 
matrix” (Huberty et al., 1987, p. 309). LCFs are used only when the population covariances are 
thought to be equal; “otherwise, quadratic classification functions (QCFs) may be used” (Huberty 
& Lowman, 1997, p. 766). In other words, the phrases “linear” rule or “linear” weights merely 
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mean that the weights are computed using the pooled variance-covariance matrix on the response 
variables, while the phrases “quadratic” rule or weights merely mean that the weights are 
computed using the separate variance-covariance matrix on the response variables for each group 
in the analysis (see Young, 1993, for more details). 

In PDA, cases receive an LCF score on each LCF by applying the LCF weights to the 
response variables. Thus, since the number of LCF’s equals the number of groups, each case has 
a number of LCF scores equal to the number of groups. In PDA, each case is assigned to the 
group corresponding to the LCF on which the case has the highest score (e g., if there are three 
groups, and case “A” had LCF scores of 1 .97, .05, and .35, respectively, that case would be 
predicted to belong to group number one). 

Classification into a certain group can also be discussed in terms of probabilities. One 
important probability to look at is prior probabilities. Huberty (1994) points out that the 
"goodness [of a classification rule], ..depends on the size (and representativeness) of the... original 
samples..." (p. 47). If one group comes from a large population, then it is more likely that a 
sample will be drawn from it. Huberty (1994) likens priors to a base rate. Huberty and Lowman 
(1997) reported that "[t]here are three types of prior probabilities...: equal priors, proportional 
priors [based on the sample size]..., and specified priors [set by the researcher ahead of time]" (p. 
765). All of these options are available in SPSS. The data in this paper were run using 
proportional priors, which presumes that in the population the groups are not equal in size. 

Once prior probabilities are taken into consideration, unit analyses can be done to 
determine posterior probabilities to decide upon group membership. The basic premise behind 
posterior probabilities is to determine the probability of group membership given one's scores on 
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the response variables. The unit is placed in the group which yields the highest posterior 
probability (Huberty, 1994). As Huberty (1994) points out, "SPSS DISCRIMINANT yields only 
the two largest [posterior probability] values" (p. 74). However, since probabilities always sum 
to 1.0, if there are three groups, the missing probability can be easily calculated by subtracting the 
two printed probabilities from 1 .0. These values can be found in Table 2, and were also removed 
from the original printout in Table 1 . 



Insert Table 2 about here 



Insight into classification can be expanded by analyzing “fence riders” and “outliers”. 
Outliers are determined by looking at the typicality probabilities. Typicality probabilities ask the 
question, given placement in group 2, "[t]o what extent does that unit 'look' like the typical 
member of Group 2" (Huberty & Lowman, 1997, p. 769)? Huberty (1994) cautions that an 
outlier "may not belong to any group..." (p. 76). Fence riders are those units which split their 
posterior probabilities evenly between two groups. We would have less confidence in our 
classification results with too many fence riders, as "[f]or each of these units, it is nearly as 
probable that they belong to one group as it is that they belong to another group" (Huberty, 
1994). 

In addition to looking at individual units, it is important to assess PDA group results in a 
classification table, an example of which can be found in Table 3. Table 3 shows a classification 
table based on the Holzinger and Swineford (1939) data, predicting grade level from student 
responses on 24 measures. The percentages in the table represent our success at prediction, or 
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our hit rate. The hit rate is the focus of the PDA analysis. It answers the question, how well did 



we predict? This question can be answered in terms of overall classification and classification by 
group. 



Insert Table 3 about here 



Methods of assessing predictive accuracy are discussed quite thoroughly in Huberty, 
Wisenbaker and Smith (1987). These authors describe three types of hit rates, which are also 
discussed in Huberty (1994). “Optimal” hit rates are "based on known parameters... applied to the 
population" (Huberty et al., 1987, p. 310). An “actual” hit rate is a rule which is applied to 
future samples, and the “expected” hit rate is the hit rate when a rule is applied to all possible 
samples (Huberty et al., 1987). 

These probabilities, or hit rates, are estimated through an “internal” analysis or an 
“external” analysis. An internal analysis uses the data at hand to obtain a classification rule. This 
rule is then applied to the data and results are given in a classification table, as mentioned above 
(Huberty & Lowman, 1997). Huberty (1994) and Huberty and Wisenbaker (1992) emphasize 
that internal analyses provide positively biased hit rates, and that a better technique would be an 
external analysis, where classification rules based on one set of data are applied to a different set 
of data for accuracy assessment. Two external analyses, outlined in Huberty (1994) and Huberty 
and Lowman (1997), are the holdout method and the leave-one-out (L-O-O) method. These 
methods use future data to assess predictive accuracy. Both external methods are available in 
SPSS, however, the L-0-0 method must be added directly to the syntax. Subcommands are 
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available in Huberty and Lowman (1997). 

Descriptive Discriminant Analysis 

Describing group differences is the job of what has been called descriptive discriminant 
analysis (DDA). We are not concerned with predicting a unit’s membership in a group; this is now 
taken as known. Instead, DDA is used to describe the effects of group membership, for example, 
to further explain the effects found in a MANOVA analysis. This paper assumes that the reader is 
familiar with MANOVA. In brief, classical MANOVA uncovers overall or “omnibus” differences 
between two groups. If statistically significant omnibus effects are detected, DDA is then used to 
detect (a) which groups differ and (b) on which response variable the groups differ. In a distance 
sense, different groups have centroids that are farther apart. DDA teases out thee differences. 

Said in another way, MANOVA assesses effects while DDA describes them (Huberty, 1994). 
Parenthetically, it might be noted that a one-way MANOVA yields exactly the same results as 
DDA as regards the omnibus test of groups differences on the response variables, but the DDA 
also simultaneously provides the more detailed information about (a) which groups differ and (b) 
on which response variables they differ. Thus, the prudent researcher with a one-way MANOVA 
problem would never do a MANOVA followed by a DDA, and would always move straight away 
to DDA. 

In the case of DDA, the response variables are the criteria or the dependent variables. 

This is opposite from the logic in PDA. In DDA, we are now focusing on the groups and how 
they differ on a set of response variables, which are combined to compute a linear discriminant 
function (LDF). The response variables are combined in a such a way that the response variable 
differences between the groups is maximized. One difference between PDA and DDA involves 
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the number “rules” (“functions”, “factors”, or “equations”); in the PDA the number of LCF rules 
(i.e., sets of weights) equals the number of groups, while in DDA the number of LDF rules equals 
the smaller of (a) the number of response variables or (b) the number of groups minus 1 . 

The focus in DDA, therefore, is on maximum group separation. What is the best way to 
describe the differences between groups of data cases? Here the grouping variable is the 
independent variable (Huberty & Barton, 1989). In this context DDA is considered a post hoc 
analysis after statistically significant results have been realized in a MANOVA analysis (Huberty, 
1984). A DDA research question, then, for Holzinger and Swineford (1939) may be, “how can 
we best describe the response variable differences between seventh and eighth grade students?”. 

The SPSS LDF weights for this analysis can be found under the heading “unstandardized 
canonical discriminant function coefficients” (Huberty & Lowman, 1997). The relevant output 
(taken from the output from Table 1) can be found for the DDA in Table 4. 



Insert Table 4 about here 



While there are more LDF’s available than LCF’s, we may not be interested in all of them. 
We are only interested in those which describe noteworthy differences between groups. Huberty 
and Lowman (1997) remind us that “[t]o best describe the grouping variable effects, it is 
important to focus only on ‘relevant’ LDF’s” (p. 774). Huberty (1994) and Huberty and Lowman 
(1997) described three ways to assess relevance: LDF plots, statistical tests, and looking at the 
“correlations between linear composite scores and scores on the individual variables in the 
composites” (Huberty, 1994, p. 209), or structure r’s (see Thompson, 1997). 
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Statistical tests use the Wilks’ A (with a corresponding F and p value) to test the null 
hypothesis that there is no dimension (“rule”, etc.) which adequately separates the groups. These 
tests are available in SPSS and an example is found in Table 2. Another strategy is to plot the 
centroids of the LDF scores (called “discriminant scores”, akin to the ubiquitous regression “yhat” 
scores). This allows to view the data in physical space to view the actual separation of the 
groups. 

Analyzing the structure r’s uses the LDF as a principal component of sorts. While there 
are other ways of finding the values of the structure r’s, Huberty (1994) recommends using the 
within-group or error structure r’s. Huberty states that “[t]he idea behind the use of the structure 
r’s is that the variables that share the most variation with a given LDF should define what the 
LDF represents” (p. 209). 

Stepwise Analysis in PDA and DDA (Both are Bad) 

While multivariate methods help us look at a number of variables at once, it is often 

necessary to identify the variable or set of variables which do the best job at predicting a certain 

outcome or describing groups differences. Thus, Cohen (1994) claimed that “we should be 

studying few independent variables and even fewer dependent variables” (p. 1304). Cohen went 

on to argue that studying too many variables is redundant, and, in this author’s opinion, stretches 

the average Ph D. student life to 10 years plus. When we do research, the key word which drives 

our efforts in research and statistical analyses is parsimony. Huberty (1989) offers the following 

definition: “The notion of parsimony in explanation or in description or in model building is 

\ 

‘scientific’” (p. 45). A goal of research should be to explain the most with the least. 

Let us now turn our discussion to variable ordering and selection. In the case of both 
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DDA and PDA, “there is a theoretical reason for retaining only ‘worthwhile’ variables or deleting 
‘worthless’ variables” (Huberty, 1994, p. 227). This concept is especially crucial to PDA, as an 
increase in variables may actually decrease our hit rate. Increasing the number of response 
variables in DDA, on the other hand, can never attenuate the uncorrected effect size (Huberty, 
1994). 

There are a number of ways to look at variable importance and order. However, one 
gross misuse of this kind is stepwise statistical analysis. Thompson (1994c) includes the use of 
stepwise procedures as one of seven common mistakes made in doctoral dissertations. Despite 
continued caution against its use (Huberty, 1989; Snyder, 1991; Thompson, 1995b), “it is quite 
common to find the use of ‘stepwise analyses’ reported in empirically based journal articles” 
(Huberty, 1994, p. 261). 

Stepwise analysis is a quick and very dirty alternative in most statistical software 
packages, and blind reliance on its use has led to erroneous conclusions in the literature. 
Essentially, forward stepwise logic is to enter a response variable into the prediction equation or 
discriminating equation one at a time, and based on a criterion, a decision is made to keep or 
remove the variable at each step. Various statistical criteria can be used in DDA, but Wilks’ 
lambda is often used. Wilks’ lambda is an inverse statistic, which means that the lower the value 
the better. 

The steps continue until all of the variables have been considered, or until adding (or 
deleting) anymore will not appreciable alter the outcome. For each lambda value there is a 
corresponding F-statistic, which provides a test for statistical significance (Huberty, 1989). The 
use of statistical significance testing has been hotly debated in the literature (Carver, 1993; Snyder 
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& Lawson, 1993), and researchers have been cautioned against the mindless use of these tests 
(Cohen, 1994). 

There are mainly three myths that naive researchers incorrectly believe are true of stepwise 
discriminant analysis. One common misconception is that stepwise procedures produce replicable 
results. The reason for this phenomena is that “stepwise methods tend to capitalize outrageously 
on sampling error” (Thompson, 1995b, p. 532). A second and related myth is that stepwise 
programs correctly select the best subset of variables. A final and pernicious myth is that 
statistical software packages are always correct, and that we should blindly accept package 
outputs as being correct. In fact, SPSS and similar packages utilize the wrong degrees of freedom 
in the calculations of stepwise statistical tests (Huberty, 1989; Snyder, 1991; Thompson, 1995b). 

A big problem with user-friendly statistical software packages is that “such convenient 
accessibility to multivariate output has led to some misuse and misinterpretation of computer 
printout information” (Huberty & Barton, 1989, p. 165). The only task that stepwise methods 
may do well is “correctly identify the best single predictor” (Thompson, 1995b, p. 532), and this 
is true only for the DDA case. An elaboration of these myths will be made concrete, again, with 
the Holzinger and Swineford (1939) data set. 

Stepwise Procedures Capitalize on Sampling Error 

“Sampling error is variability in sample data that is unique to the given sample” 

(Thompson, 1995b, p. 532), and therefore, makes replicability of results in future samples 
unlikely. In SPSS, Wilks’ A can be used for variable selection in discriminant analysis. The F- 
statistic forms the basis for decision-making. Rencher and Larson (1980) explained that, “At each 
stage the variable with the largest F-to -enter is added to the set of variables if its F-value is larger 
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than a specified threshold” (p. 350). 

It should be noted, however, that variables are entered even if their F-values are just barely 
larger than other variables. Thompson (1995b) explained, “It is entirely possible that this 
infinitesimal advantage of [one variable] over another variable is sampling error” (p. 532). If a 
variable is selected by mistake very early on, this will throw off the selection of subsequent 
variables, as the entry decisions are conditional on all previous selection decisions (Snyder, 1991). 

Rencher and Larson (1980) take this argument a step further, and claim that bias in Wilks’ 
A and the instability of using statistical significance testing as a stopping rule may lead to 
problems such as: 

Inclusion of too many variables in the subset. If the significance level shown on a 
computer output... is used as an informal stopping rule, some variables will likely be 
included which do not contribute to separation of groups. A subset chosen with 
significance levels as guidelines will not likely be stable... (p. 350) 

These authors go on to argue that the computer may very well select a completely spurious subset 
of variables, which is not detected if the researcher goes by F-values alone. This will be further 
discussed in the next section. 

Stepwise Procedures Do Not Necessarily Choose the Best Subset of Variables 

Probably the most common misconception among applied researchers is the wrong-headed 
belief that stepwise methods identify the best variable set of a given size. Stepwise methods do 
not do this. Thus, Snyder (1991 ) reminds us that, “Selecting the next-best predictor for each of q 
steps is not the same as selecting the optimal predictor variable set of size q” (p. 101). The 
problem can be made concrete by considering an example given by Thompson (1995b) using the 
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analogy of a five-member basketball team. Choosing the best five individual players does not 
necessarily ensure that one has chosen the best overall team. There may be another team of five 
which, and potentially even not made up of any of the best players, work better as a team. This 
phenomenon is illustrated using the Holzinger and Swineford (1939) data in the following 
heuristic example. 

The first step in the illustration was to tell the computer to perform a stepwise 
discriminant analysis (the syntax file is available in Appendix B). F-to-enter and F-to-remove 
parameters were set in order to limit the number of variables chosen to four. Table 5 shows a 
reproduced table from the output file indicating the variables chosen as the “best subset” were 
T10, T19, T3, and T17. The corresponding Wilks’ A , F-statistics, degrees of freedom, and p 
levels are shown. Note that the final A value is .805. The naive researcher would take these 
results, do a couple of flips, and claim golden results. Manipulation of the data, however, reveals 
serious problems. 



Insert Table 5 about here 



The next step was to systematically enter different variables available from the data set 
into a non-stepwise discriminant analysis. The results, while the process was somewhat tedious, 
were interesting. Table 6 shows that entering variables T3, T10, T12, and T19 result in the same 
A value reached in the stepwise method. While a lower A value would have been ideal for 
illustrative purposes, an interesting point can be made regarding practical problems which arise 
with stepwise methodology. 
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Insert Table 6 about here 



Indeed, it is entirely possible that a different set of four predictors might have yielded a 
smaller (i.e., better) A value. Note that the variables in Table 6 differ from those in Table 5. In 
Table 6, T12 substituted for T 17. 

T12 is a variable representing a score on a test called “speeded counting of dots in shape.” 
T 1 7 is labeled “memory of object-number association targets.” For illustrative purposes, let us 
presume that T17, the variable chosen by the stepwise procedure takes 30 minutes to administer, 
while T12 takes only 5 minutes. This is an important difference if testing fatigue is an issue. 
However, without this close scrutiny, a researcher may choose T17 over T12 solely based on 
SPSS results. This example strongly argues against blind interpretation of computer results. It 
has become far too easy for researchers to stop thinking, however, and what is easy is often 
inappropriate. 

Software Packages Use the Wrong Degrees of Freedom 

While it may seem that this paper has focused on the ills of computer software, this is 
entirely not the case. Researchers, however, need to be aware that certain software mistakes are 
also evident, and need to be avoided. In stepwise analyses, statistical packages, such as SPSS, 
consistently use the wrong value for the degrees of freedom in the F-statistic. Thompson (1995b) 
describes degrees of freedom as follows: 

Degrees of freedom in statistical analyses reflect the number of unique pieces of 
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information present for a given research situation. These degrees of freedom 
constrain the number of inquiries we may direct at our data and are the currency 
we spend in analysis, (p. 526) 

In typical programs, 1 degree of freedom explained is charged at step one of a stepwise 
analysis. In reality, however, all of the variables are being consulted for that given step. At step 
two, k-1 variables are consulted, but only two degrees of freedom explained are charged. 

Charging too little for the analysis results in a positively biased F-value. The degrees of freedom 
explained for each step should be the total number of variables since every variable is examined to 
decide which one to ultimately enter. And if the degrees of freedom for the numerator are wrong, 
then of course so are the degrees of freedom unexplained. Both these wrong degrees of freedom 
bias the calculated F statistic is the same direction: positively. Results of the computer package 
tests can be tremendously different from those employing the correct degrees of freedom. An 
example using the Holzinger and Swineford (1939) data is included in Table 7. 



Insert Table 7 about here 



The formula at the top of Table 8, F = [(1 - A) / A]* (df error / df effect) is used to 
recalculate the F-statistic. This is done for two reasons. One reason is to show the reader that 
the wrong degrees of freedom can result in spuriously high F-values. The other reason is to show 
the reader a way to correct the computer’s mistake. In the first calculation, the F-statistic is done 
the wrong way (i.e., the way SPSS would calculate it). Note that the F-value of 41.408 is almost 
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equal to the F-value calculated and found in Table 6, 41.360. This was done to check the 
formula. Underneath, the F-value is recalculated using the correct degrees of freedom. The new 
value, 0.0105347, is quite different, and would not reach statistical significance, since an F less 
than 1.0 can not be statistically significant even at infinite degrees of freedom. “This statistical 
welfare system may cause us to radically overestimate the atypicality of our results” (Thompson, 
1995b, p. 527), and again hurt any chances for replication in the future. 

Double-Idiot-Syndrome: Using Stepwise Methodology in PDA 

The final stake to drive into the heart of stepwise methodology highlights, again, the 
differences between PDA ad DDA, outlined earlier. To review, Klecka (1980) describes the 
DDA as follows. 

A researcher is engaged in interpretation when studying the ways in which groups differ- 
that is, is one able to ‘discriminate’ between the groups on the basis of some set of 
characteristics, how well do they discriminate, and which characteristics are the most 
powerful discriminators? (p. 9) 

The last part of the question describes the focus of stepwise DDA. Which variables discriminate 
the groups (i.e., describe group differences) the best? To answer these questions, stepwise 
methodology looks at Wilks’ A and the F-statistic. These values are used to decide which 
variables “ do the best job”. 

Remember, however, that PDA is not concerned with group differences, but solely and 
exclusively with classification. The focus is on how well the discriminant function can predict 
group membership. Huberty (1994) states that. “A hit results when a unit emanating from group 
g is assigned (by means of the classification rule used) to group g” (p. 78). The hit rate is 
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computed by dividing the hits by the total n of that group. When assessing the performance of 
variables as a subset, therefore, the researcher looks to the classification table which indicates hit 
rate for both groups, and or each group individually. An example of a classification table is found 
in Table 8. But stepwise methodology focuses on group separation, NOT hit rate. In fact, it 
should be noted that PDA prediction accuracy can actually go down with more predictors, unlike 
DDA where more response variables cannot hurt (increase) the value of Wilks’s A (Huberty, 
1984). DDA stepwise, therefore, is completely pointless when conducting DA for PDA 
classification purposes. 



Insert Table 8 about here 



Using the heuristic data from Holzinger and Swineford (1939), it was actually quite easy 
to find a set of predictors which yielded a better hit rate than the stepwise procedure. In addition, 
the results affected each group differently, which has practical implications as well. Table 9 
contains the classification result from the stepwise output partially reproduced in Table 6. The 
total hit rate was found to be 69.8%. Substituting variable T9 for T1 7, the new total hit rate was 
higher: 70.4%. When one looks at the individual hit rates, the differences are even greater. For 
prediction in the eighth grade population, the hit rate went up almost 3 percentage points. This is 
important if the researcher is specifically interested in this population. This becomes especially 
important in high-stakes decision-making, like promotion or retention situations. I am sure that a 
potential eighth grader would feel more comfortable using the scores on the variables that I chose, 
than on those the computer stepwise program chose, if passing onto the eighth grade hung in the 
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balance. 



Insert Table 9 about here 



Alternatives to Stepwise Methodology 

When choosing the best subset of variables is important, a few alternatives are available, 
although not necessarily in computer packages such as SPSS. The first alternative, if you have 
the time, is to painstakingly substitute other variables into a discriminant analysis to arrive at 
possibly a better hit rate or a lower Wilks’ A value, as seen in the heuristic data set. 

Other alternatives are also available. Some specialized computer programs can also be 
used for purposes of variable selection. For example, for DDA cases the program offered by 
McCabe (1975) may be used. For PDA cases, the Morris program distributed within Huberty’s 
(1994) book, or related programs by Morris described elsewhere (cf. Morris & Huberty, 1995). 
Huberty (1989) encouraged the use of these alternatives keeping the following in mind: 

(1) the collection of variables to be studied constitutes a meaningful system from 
a substantive standpoint; (2) the size of the initial variable system is manageable, 
that is, the number of response variables is about 30 or less; (3) there is a 
theoretical or substantive or practical reason for wanting to delete some 
variables; and (4) the researcher has access to a computer and some computer 
package software, (p. 49) 

One alternative to stepwise which specifically looks at variable ordering considers using a 
partial F (i.e., considering the contribution of given variable partialling out the other variables) 
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(Huberty, 1989). Huberty goes on to outline other alternatives as well, including cross-validation 
procedures, acknowledging the need to utilize only those variables which fit theoretically, 
practically, and add to the predictive validity of our solution. 

Some of the problems (i.e., wrong degrees of freedom) can be adjusted by hand, as noted 
in the previous sections. Just knowing that a problem exists leads to better alternatives and better 
analytical judgement. In conclusion, the bottom line seems to be that researchers need to learn 
how to think for themselves, and abstain from blind use of computer programs. 
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Table 1 

Output Discriminant Analysis 



Discriminant 



Group Statistics 







Valid 


N (listwise) 


TRACK 




Unweighted 


Weighted 


JUNE PROMOTIONS 


T22 


219 




219.000 






T17 


219 


219.000 






T5 


219 


219.000 


FEB PROMOTIONS 




T22 


82 


82.000 






T17 


82 


82.000 






T5 


82 


82.000 


Total 


T22 


301 




301.000 






T17 


301 


301.000 






T5 


301 


301.000 



Summary of Canonical Discriminant Functions 

Eigenvalues 

Function Eigenvalue % of Variance Cumulative % Canonical 

Correlation 

1 .071(a) 100.0 100.0 .257 

a. First 1 canonical discriminant functions were used in the analysis. 

Wilks 1 Lambda 

Test of Function(s) Wilks’ Lambda Chi-square df Sig. 

1 .934 20.315 3 .000 

Standardized Canonical Discriminant Function Coefficients 

Function 1 
T22 .265 

T17 -.457 

T5 .794 



Structure Matrix 

Function 1 
T5 .876 

T22 .517 

T17 -.367 

Pooled within-groups correlations between discriminating variables and 
standardized canonical discriminant functions 

Variables ordered by absolute size of correlation within function. 

Functions at Group Centroids 

TRACK Function 1 

JUNE PROMOTIONS .162 
FEB PROMOTIONS -.433 

Unstandardized canonical discriminant functions evaluated at group means 
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Classification Statistics 



Prior Probabilities for Groups 
Prior Cases Used in Analysis 



TRACK 

JUNE PROMOTIONS 


.728 


Unweighted 

219 


Weighted 

219.000 




FEB PROMOTIONS 


.272 


82 


82.000 




Total 


1.000 


301 


301.000 






Classification Results (a) 

Predicted Group Membership 
TRACK JUNE 


FEB 


Total 


Original Count 


JUNE 


PROMOTIONS 


PROMOTIONS 






PROMOTIONS 


211 


8 


219 




FEB 

PROMOTIONS 


74 


8 


82 


% 


JUNE 

PROMOTIONS 


96.3 


3.7 


100.0 




FEB 

PROMOTIONS 


90.2 


9.8 


100.0 



a 72.8% of original grouped cases correctly classified. 
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Table 2 



Posterior Probabilities 



Illustrative Results for the First 50 Cases 



Case 


Mis 


Actual 




Highest 


Probability 


2nd Highest 


Discrim 


Number 


Val Sel 


Group 




Group P 


(D/G) 


P ( G/D ) 


Group 


P ( G/D ) 


Scores 


1 




2 


* * 


1 


.8160 


. 5782 


2 


.4218 


.3948 


2 




2 




2 


.6146 


.6170 


1 


.3830 


-.9365 


3 




2 




2 


.6270 


.6145 


1 


.3855 


-.9190 


4 




2 


* * 


1 


.9137 


. 5601 


2 


.4399 


.2704 


5 




2 




2 


.5196 


. 6365 


1 


.3635 


-1.0770 


6 




2 




2 


.7741 


. 5861 


1 


.4139 


-.7201 


7 




2 


★ * 


1 


.9863 


. 5416 


2 


.4584 


. 1449 


8 




2 




2 


.4778 


.6456 


1 


. 3544 


-1.1428 


9 




2 




2 


. 1393 


. 7421 


1 


. 2579 


-1.9113 


10 




2 




2 


.8421 


. 5146 


1 


.4854 


-.2338 


11 




2 




2 


.9846 


. 5470 


1 


.4530 


-.4523 


12 




2 


★ * 


1 


.7752 


. 5018 


2 


.4982 


-.1234 


13 




2 




2 


.8670 


. 5688 


1 


.4312 


-.6005 


14 




2 




2 


.8202 


. 5775 


1 


.4225 


-.6603 


15 




2 


* * 


1 


.2128 


. 7147 


2 


.2853 


1.4080 


16 




2 




2 


.9005 


. 5626 


1 


.4374 


-.5581 


17 




2 




2 


.4988 


.6410 


1 


.3590 


-1.1093 


18 




2 




2 


.6665 


.6067 


1 


. 3933 


-.8640 


19 




2 


★ * 


1 


.0195 


.8274 


2 


. 1726 


2.4981 


20 




2 


* * 


1 


.6116 


.6176 


2 


. 3824 


. 6700 


21 




2 




2 


.3283 


. 6811 


1 


. 3189 


-1.4106 


22 




2 




2 


.4797 


.6451 


1 


.3549 


-1.1397 


23 




2 




2 


.9471 


. 5344 


1 


.4656 


-.3667 


24 




2 




2 


.9266 


. 5305 


1 


.4695 


-.3409 


25 




2 


★ * 


1 


.9465 


. 5342 


2 


.4658 


.0950 


26 




2 


* * 


1 


. 7367 


. 5932 


2 


.4068 


.4983 


27 




2 




2 


.1422 


.7409 


1 


.2591 


-1.9007 


28 




2 




2 


. 5392 


.6324 


1 


.3676 


-1.0470 


29 




2 




2 


. 5869 


.6226 


1 


. 3774 


-.9764 


30 




2 




2 


.7417 


. 5922 


1 


.4078 


-.7626 


31 




2 




2 


. 7013 


.6000 


1 


.4000 


-.8165 


32 




2 




2 


.9355 


. 5561 


1 


.4439 


-.5139 


33 




2 


★ * 


1 


.2551 


.7015 


2 


.2985 


1.3003 


34 




2 




2 


.8872 


. 5232 


1 


.4768 


-.2912 


35 




2 




2 


.4792 


.6453 


1 


. 3547 


-1.1407 


36 




2 




2 


.8387 


.5140 


1 


.4860 


-.2295 


37 




2 




2 


.9841 


. 5412 


1 


. 4588 


-.4131 


38 




2 




2 


.7761 


.5020 


1 


.4980 


-.1486 


39 




2 




2 


.8923 


. 5241 


1 


.4759 


-.2975 


40 




2 


** 


1 


.0576 


.7870 


2 


.2130 


2.0607 


41 




2 




2 


.6082 


.6183 


1 


. 3817 


-.9457 


42 




1 


★ * 


2 


. 7666 


. 5001 


1 


.4999 


-.1362 


43 




1 


★ * 


2 


.9711 


. 5495 


1 


.4505 


-.4692 


44 




1 


** 


2 


.6646 


.6071 


1 


.3929 


-.8666 


45 




1 




1 


.7664 


. 5876 


2 


.4124 


.4592 


46 




1 


* * 


2 


.7029 


. 5997 


1 


.4003 


-.8144 


47 




1 


* * 


2 


.6761 


.6048 


1 


. 3952 


-.8507 


48 




1 


* * 


2 


. 6305 


.6138 


1 


.3862 


-.9140 


49 




1 


★ * 


2 


.9528 


.5529 


1 


.4471 


-.4922 


50 




1 




1 


.6456 


.6108 


2 


.3892 


. 6220 
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Table 3 



Classification Table 



Classification Results (a) 







Predicted 


Group Membership 










TRACK 


JUNE 


FEB 


Total 


Original 


Count JUNE 


PROMOTIONS 


PROMOTIONS 








PROMOTIONS 


211 


8 


219 






FEB 












PROMOTIONS 


74 


8 


82 




% 


JUNE 












PROMOTIONS 


96.3 


3.7 


100.0 






FEB 












PROMOTIONS 


90.2 


9.8 


100.0 


a 72.8% 


of 


original grouped cases correctly classified. 
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Table 4 



Standardized Canonical Discriminant Function Coefficients 

Standardized Canonical Discriminant Function Coefficients 

Function 1 
T22 .265 

T17 -.457 

T5 .794 
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Table 5 

Output Table from Stepwise Discriminant Analysis 



Step 


Variable 


Wilks’ A 


F-statistic 


df 


p-value 


1 


T10 


0.878 


41.360 


1 


0.000 


2 


T19 


0.839 


28.676 


2 


0.000 


3 


T3 


0.815 


22.539 


3 


0.000 


4 


T17 


0.805 


17.948 


4 


0.000 
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Table 6 

Discriminant Analysis Output Substituting T12 for T17 



Function Wilks’ A Chi-square df p-value 



1 



0.805 



64.338 



4 



0.000 
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Table 7 

Correcting for the Wrong Degrees of Freedom 



F = [(1 - A) / A]* (df error / df effect) 
STEP ONE: [(1-0.878) / 0.878] * (298 / 1) 

[0.122/0.878] *298 
[.01389522] * 298 
41.408 

STEP ONE: [(1-0.878) / 0.878] * (277/21) 

[ 0.122/0.878] * 13.19 
[ 01389522] * 13.19 



0.0105347 
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Table 8 

Classification Table for Stepwise Output 







Predicted Group 






7 


8 


Original Group 


7 


118 

(75.2%) 


39 

(24.8%) 




8 


52 

(36.1%) 


92 

(63.9%) 



69.8% of original grouped cases correctly classified 
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Table 9 

Classification Table Substituting T9 for T17 







Predicted Group 






7 


8 


Original Group 


7 


117 

(74.5%) 


40 

(25.5%) 




8 


49 

(34%) 


95 

(66%) 




70.4% of original grouped 


cases correctly classified 
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Appendix A 



SET BLANKS=SYSMIS UNDEFINED=WARN printback=list . 

TITLE ' MULTINOR . SPS tests multivar normality graphically****'. 
COMMENT * ******************************************************** 
COMMENT The original MULTINOR computer program was presented, 
COMMENT with examples, in: 

COMMENT Thompson, B. (1990). MULTINOR: A FORTRAN program that 

COMMENT assists in evaluating multivariate normality. 

COMMENT ^Educational and Psychological Measurement^, 50, 

COMMENT 845-848. 

COMMENT 

COMMENT The logic and the data source for the example are from: 
COMMENT Stevens, J. (1986). ^Applied multivariate statistics 

COMMENT for the social sciences^. Hillsdale, NJ: Erlbaum. 

COMMENT (pp. 207-212) 

COMMENT *********************************************************** 
COMMENT Here there are 3 variables for which multivariate 
COMMENT normality is being confirmed. 

DATA LIST 



FILE= ' c : \spsswin\multinor . dat ' FIXED RECORDS=l TABLE 
/I xl 1-3 (1) x2 5-7 (1) x3 9-11 (1). 

list variables=all/cases=9999/f ormat=numbered . 

COMMENT 'y' is a variable automatically created by the program, and 
COMMENT does not have to modified for different data sets, 
compute y=$casenum . 
print formats y(F5) . 

regression variables=y xl to x3/ 
descriptive=mean stddev corr/ 
dependent=y/enter xl to x3/ 
save=mahal (mahal ) . 

sort cases by mahal(a) . 
execute . 

list variables=y xl to x3 mahal/cases=9999/f ormat=numbered . 

COMMENT In the next TWO lines, for a given data set put the actual n 
COMMENT in place of the number '12' used for the example data set. 
loop #i=l to 12 . 

COMMENT In the next line, change *3' to whatever is the number 
COMMENT of variables. 

COMMENT The p critical value of chi square for a given case 

COMMENT is set as [the case number (after sorting) - .5] / the 

COMMENT sample size] . 

compute p=($casenum - .5) / 12 . . 

compute chisq=idf . chisq (p, 3 ) . 

end loop . 

print formats p chisq (F8.5) . 

list variables=y p mahal chisq/cases=9999/f ormat=numbered . 
plot 

vertical= ’ chi square'/ 

hori zontal= ' Mahalabis distance'/ 

plot=chisq with mahal . 
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Appendix B 



IF 


(ID 


GT 


IF 


(ID 


GE 


IF 


(ID 


GE 


IF 


(ID 


GE 


IF 


(ID 


GE 


IF 


(ID 


GE 


IF 


(ID 


GE 


IF 


(ID 


GE 


IF 


(ID 


GE 



FILE=' c:\prograra f iles\spss\coursework\HOLZINGR . DTA' FIXED RECORDS=2 TABLE 
/I id 1-3 sex 4-4 ageyr 6-7 

agerao 8-9 tl 11-12 t2 14-15 t3 17-18 t4 20-21 t5 23-24 t6 26-27 t7 29-30 t8 
32-33 t9 35-36 tlO 38-40 til 42-44 tl2 46-48 tl3 50-52 tl4 54-56 tl5 58-60 
tl6 62-64 tl7 66-67 tl8 69-70 tl9 72-73 t20 74-76 t21 78-79 /2 t22 11-12 
t23 14-15 t24 17-18 t25 20-21 t26 23-24 . 

EXECUTE. 

COMPUTE SCHOOL=l. 

(ID GT 200) SCHOOL=2 . 

(ID GE 1 AND ID LE 85)GRADE=7. 

(ID GE 86 AND ID LE 168)GRADE=8. 

(ID GE 201 AND ID LE 281)GRADE=7. 

(ID GE 282 AND ID LE 351)GRADE=8. 

(ID GE 1 AND ID LE 44)TRACK=2. 

(ID GE 45 AND ID LE 85)TRACK=1. 

(ID GE 86 AND ID LE 129)TRACK=2. 

(ID GE 130) TRACK=1 . 

PRINT FORMATS SCHOOL TO TRACK (FI. 0). 

VALUE LABELS SCHOOL ( 1 ) PASTEUR (2) GRANT-WHITE/ 

TRACK ( 1 ) JUNE PROMOTIONS (2) FEB PROMOTIONS/. 

VARIABLE LABELS Tl VISUAL PERCEPTION TEST FROM SPEARMAN VPT, PART III 
T2 CUBES, SIMPLIFICATION OF BRIGHAM'S SPATIAL RELATIONS TEST 
T3 PAPER FORM BOARD — SHAPES THAT CAN BE COMBINED TO FORM A TARGET 
■ T4 LOZENGES FROM THORNDIKE — SHAPES FLIPPED OVER THEN IDENTIFY TARGET 
T5 GENERAL INFORMATION VERBAL TEST 
T6 PARAGRAPH COMPREHENSION TEST 
T7 SENTENCE COMPLETION TEST 

T8 WORD CLASSIFICATION — WHICH WORD NOT BELONG IN SET 

T9 WORD MEANING TEST 

TlO SPEEDED ADDITION TEST 

Til SPEEDED CODE TEST — TRANSFORM SHAPES INTO ALPHA WITH CODE 

T12 SPEEDED COUNTING OF DOTS IN SHAPE 

T13 SPEEDED DISCRIM STRAIGHT AND CURVED CAPS 

T14 MEMORY OF TARGET WORDS 

T15 MEMORY OF TARGET NUMBERS 

T16 MEMORY OF TARGET SHAPES 

T17 MEMORY OF OBJECT-NUMBER ASSOCIATION TARGETS 

T18 MEMORY OF NUMBER-OBJECT ASSOCIATION TARGETS 

T19 MEMORY OF FIGURE-WORD ASSOCIATION TARGETS 

T20 DEDUCTIVE MATH ABILITY 

T21 MATH NUMBER PUZZLES 

T22 MATH WORD PROBLEM REASONING 

T23 COMPLETION OF A MATH NUMBER SERIES 

T24 WOODY -MCCALL MIXED MATH FUNDAMENTALS TEST 

T25 REVISION OF T3 — PAPER FORM BOARD 

T26 FLAGS — POSSIBLE SUBSTITUTE FOR T4 LOZENGES. 



DISCRIMINANT 

/ GROUPS=grade ( 7 8) 

/VARIABLES=tl tlO til tl2 tl3 tl4 tl5 tl6 tl7 tl8 tl9 t2 t20 t21 t22 t23 
t24 t25 t26 t3 t4 t5 t6 t7 t8 t9 
/ANALYSIS ALL 
/PRIORS SIZE 

/ STATISTICS=RAW CORR COV GCOV CROSSVALID 
/ CLASSIFY=NONMISSING POOLED . 
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DISCRIMINANT 

/GROUPS=grade (7 8) 

/VARIABLES=tlO til tl2 tl3 tl4 tl5 tl6 tl7 tl8 tl9 t20 t21 t22 t23 t24 t3 t4 
t5 t6 t7 t8 t9 
/ANALYSIS ALL 
/METHOD=WILKS 
/ FIN= 3.5 
/ FOUT= 2.71 
/PRIORS SIZE 
/HISTORY 

/STATISTICS=TABLE 
/ CLASSIFY=NONMISSING POOLED . 

DISCRIMINANT 

/GROUPS=grade (7 8) 

/VARIABLES=t3 tlO tl2 tl9 
/ANALYSIS ALL 
/PRIORS SIZE 
/STATISTICS=TABLE 
/ CLASSIFY=NONMISSING POOLED . 
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